A real-time depth camera is able to determine the distance to a human or other object in a field of view of the camera, and to update the distance substantially in real time based on a frame rate of the camera. Such a depth camera can be used in a motion capture systems, for instance, to obtain data regarding the location and movement of a human body or other subject in a physical space, and can use the data as an input to an application in a computing system. Many applications are possible, such as for military, entertainment, sports and medical purposes. A depth camera typically includes an infrared illuminator which illuminates the field of view, and one or more infrared image sensors which sense reflected light from the field of view to form an image. Moreover, the depth camera can be provided as part of a gaming console which communicates with a display device such as a television in a user's home. However, various challenges exist in adapting depth camera technology for portable or other small form factor devices such a video projector devices. These challenges include power and space limitations.